Imaging apparatus and reproducing apparatus which changes frame rate based on zoom operation

ABSTRACT

An imaging unit outputs a video signal at a frame rate higher than a standard frame rate. During a zoom operation period or a period including the zoom operation period and periods before and after the zoom operation, a video signal from the imaging unit is recorded in a recording medium at a high recording frame rate. Other than this period, a video signal is recorded in the recording medium at the standard frame rate. The recording frame rate and zoom operation information is recorded as metadata in the recording medium. During reproduction, based on a set reproduction mode, thinning processing is carried out on a video signal recorded during a zoom operation, and the processed signal is output at the standard frame rate. In this way, it is possible to change a frame rate in view of a photographer&#39;s intention and assure compatibility with existing viewing and reproduction environments.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Pat. No. 8,508,627 filed Sep.6, 2012, which is a continuation of U.S. Pat. No. 8,264,573 filed May11, 2010, which claims the benefit of and priority to Japanese PatentApplication No. 2009-121535 filed May 20, 2009, the contents of each ofwhich are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging apparatus that recordsmoving images, and to a reproducing apparatus that reproduces the movingimages, such as a video camera or a digital camera.

2. Description of the Related Art

In recent years, a recording and reproducing apparatus capable ofrecording/reproducing high-quality video signals such as high definition(HD) video signals or HD signals, and a video display apparatus capableof displaying such signals as images have been widely spread. Backgroundtechniques of an image processing system used in these apparatusesinclude techniques of miniaturizing the size of an image sensor,increasing the number of pixels, increasing the speed of transferprocessing, and improving the efficiency of coding such as the H.264standard. In addition, various types of video display apparatuses havebeen made available, such as those of a liquid crystal type, a plasmatype, and an electro-luminous (EL) type. Further, video displayapparatuses having a further decreased thickness, a higher definition,and/or a larger screen are being developed. Mobile-type video displayapparatuses are also being developed.

Since display apparatuses of such various types have been becomeavailable, users' viewing styles have changed significantly. Usershandle images in various manners depending on the purpose, for example,some users prefer high-quality images while others wish to exchangeshort movies with ease.

Techniques for storing large amounts of video data play an importantrole in addressing the needs of users. Particularly, remarkable progresshas been made in the increase of the capacity of hard disks and in thedecrease of costs thereof, followed by semiconductor memories.Additionally, large-capacity removable media or servers on the order ofterabytes have been made available at low cost.

Under the background described above, while the frame rate of thenational television system committee (NTSC) is 29.97 frames/second inthe field of imaging techniques, higher frame rates such as 120frames/second and 240 frames/second are proposed. Further, techniquesfor multi-screen reproduction and reproduction with multi-audio channelsare also being developed. Based on these techniques, more realistic anddynamic images can be displayed.

While video signals are conventionally recorded at a fixed frame rate,some techniques are proposed for appropriately changing the frame rate,efficiently suppressing the total amount of recording data, andrecording high-quality videos when necessary.

Japanese Patent Application Laid-Open No. 2003-274360 (U.S. Pat. No.7,456,875) discusses a technique for controlling the frame rate ofcaptured video signals without requiring a photographer's control. Morespecifically, based on this technique, when an image is captured, asurrounding audio volume level is monitored, and if it is detected thatthe monitored volume level exceeds a predetermined level, the frame rateis increased, so that important scenes are recorded with high quality.Further, based on this technique, when the available memory capacity ofa recording medium reaches less than or equal to a predetermined level,the frame rate is decreased to suppress consumption of the recordingmedium. In addition, when the available power of a battery reaches lessthan or equal to a predetermined level, the frame rate is decreased tosuppress consumption of the battery.

Japanese Patent Application Laid-Open No. 2007-134991 (USPA2007/0104462) discusses a technique in which motion of a captured imageis detected. Based on this technique, when the detected motion is morethan or equal to a threshold, the captured image is recorded at a highframe rate, and when the detected motion is less than the threshold, thecaptured image is recorded at a low frame rate.

However, based on the above techniques discussed in Japanese PatentApplication Laid-Open No. 2003-274360 (U.S. Pat. No. 7,456,875) andJapanese Patent Application Laid-Open No. 2007-134991 (USPA2007/0104462), change of the frame rate does not reflect photographer'sintention. Namely, based on the technique discussed in Japanese PatentApplication Laid-Open No. 2003-274360 (U.S. Pat. No. 7,456,875), theframe rate is changed depending on conditions surrounding an imagingapparatus, such as the surrounding audio volume level or the availablepower of a recording medium or a battery. Thus, the imaging condition ischanged without photographer's intention.

In addition, the technique discussed in Japanese Patent ApplicationLaid-Open No. 2007-134991 (USPA 2007/0104462) uses motion informationacquired from an image signal as an index. For example, when the ratioof monotonous scenes including the sky or night scenes is increased, itis determined that the amount of motion is less, and as a result, theimage is recorded at a low frame rate. On the other hand, in a wideimage whose focal length is short, the motion amount tends to bedetected as being large because of camera shake, and consequently, theimage is recorded at a high frame rate. In either case, the frame rateis determined irrespective of photographer's intention.

Normally, TV receivers or video monitors support only a fixed framerate. Thus, when a video signal is input and the frame rate thereof ischanged in the middle of processing, even if the change of the framerate can be followed, the video may be displayed at an unnaturalrendering speed misaligned with the actual time. For example, if amonitor receives an input video signal and displays the signal at acertain rate images captured at a frame rate higher than the standardframe rate are rendered in slow motion, and images captured at a lowerframe rate are rendered at high speed.

Since compatibility between these viewing and reproduction environmentsis not taken into consideration, use of the technique of variable framerate recording is limited.

SUMMARY OF THE INVENTION

The present invention is directed to an imaging apparatus capable ofchanging a frame rate in view of a photographer's intention andmaintaining compatibility with existing viewing and reproductionenvironments.

According to an aspect of the present invention, an imaging apparatusincludes an imaging unit including an image sensor configured to convertan optical image to an image signal, and a zoom unit configured tooptically zoom an optical image incident on the image sensor orelectronically zoom an image signal output from the image sensor, arecording unit configured to record a video signal including an imagesignal captured by the imaging unit in a recording medium, andconfigured to record a recording frame rate, at which the video signalis recorded in the recording medium, and zoom operation information forthe zoom unit in the recording medium, a zoom operation unit configuredto operate the zoom unit, a control unit configured to control therecording frame rate at which the video signal is recorded in therecording medium based on an operation of the zoom operation unit, andconfigured to increase the recording frame rate to be higher than anormal frame rate during a period including a period when the zoomoperation unit is operated, and a reproducing unit configured toreproduce the video signal from the recording medium based on a setreproduction mode, and configured to carry out thinning processing onthe video signal during the period including the period when the zoomoperation unit is operated based on the zoom operation information andreproduce the processed video signal at the normal frame rate.

Based on an imaging apparatus and a reproducing apparatus according tothe present invention, videos including zoom operations can bereproduced. Since videos are reproduced at a standard frame rate,compatibility with existing video display apparatuses can be ensured.During periods including zoom operation periods, videos are recorded ata frame rate higher than a normal frame rate, and when the videos arereproduced, the frame rate is reduced to the standard frame rate. Thus,high-quality reproduction videos can be displayed.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram illustrating a schematic configuration of anexemplary embodiment of the present invention.

FIGS. 2A to 2C illustrate a relationship among a zoom operation,captured frames, and recorded frames of the exemplary embodiment.

FIG. 3 is a flow chart illustrating a recording operation of theexemplary embodiment.

FIG. 4 illustrates a relationship between a zoom speed and a recordingframe rate of the exemplary embodiment.

FIG. 5 illustrates a metadata structure and data examples of the presentexemplary embodiment.

FIG. 6 illustrates a relationship between zoom magnification andrecorded frames of the exemplary embodiment.

FIG. 7 illustrates recording areas each storing frames at differentframe rates.

FIG. 8 is a flow chart illustrating a reproduction operation of theexemplary embodiment.

FIG. 9 is a table illustrating reproduction modes that involve zoomoperations.

FIG. 10 illustrates frame interpolation in reproduction mode 1 of theexemplary embodiment.

FIG. 11 illustrates frame interpolation in reproduction mode 2 of theexemplary embodiment.

FIG. 12 illustrates frame interpolation in reproduction mode 3 of theexemplary embodiment.

FIG. 13 illustrates frame interpolation in reproduction mode 5 of theexemplary embodiment.

FIG. 14 is a block diagram illustrating functions of a system controlunit 50.

FIGS. 15A to 15D illustrate a relationship among a zoom operation,captured frames, and recorded frames of a second exemplary embodiment.

FIG. 16 illustrates a relationship between zoom magnification andrecorded frames of the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 is a block diagram illustrating a schematic configuration of anexemplary embodiment of the present invention. A configuration and abasic operation of the present exemplary embodiment will be describedwith reference to FIG. 1.

An imaging unit 10 (camera unit) has the following configuration andfunctions. The imaging optical system of the imaging unit 10 includes afront lens 12, a zoom lens 14, a diaphragm 16, and a focus lens 18. Thefront lens 12 is fixed to a lens barrel, and the zoom lens 14 and thefocus lens 18 can move along the optical axis. The diaphragm 16 isarranged between the zoom lens 14 and the focus lens 18.

An exposure control unit 20 receives a control signal from a cameracontrol unit 30 and controls the aperture of the diaphragm 16 based onthe control signal. A zoom position sensor 22 detects the position ofthe zoom lens 14. Based on the position of the zoom lens 14 detected bythe zoom position sensor 22, a lens control unit 24 carries out feedbackcontrol on the zoom lens 14. The lens control unit 24 also controls thezoom position of the zoom lens 14 and the focus position of the focuslens 18, based on instruction signals from the camera control unit 30 tobe described later.

An image sensor driving unit 28 drives an image sensor 26 that convertsan optical image formed by the imaging optical system into an electricimage signal. The image sensor driving unit 28 drives the image sensor26 based on a timing signal supplied from the camera control unit 30 andcontrols the image sensor 26 to output each pixel signal for the imagesignal. By increasing the driving signal frequency of the image sensordriving unit 28, a frame rate higher than a standard frame rate can behandled.

A sample-and-hold (S/H) unit 32 amplifies the image signal output fromthe image sensor 26, and samples and holds the signal based on a timingsignal supplied from the camera control unit 30 (or the image sensordriving unit 28). An analog-to-digital (A/D) converter 34 converts thesignal supplied from the S/H unit 32 into a digital signal, and suppliesthe digital signal to a camera signal processing unit 36.

Based on parameters set by the camera control unit 30, the camera signalprocessing unit 36 carries out known camera signal processing, such ascolor separation, gradation correction, and white balance adjustment, onthe image data output from the A/D converter 34. The camera signalprocessing unit 36 supplies the processed image data to acompression/expansion unit 44 via a data bus 38 and a memory 40.

While an audio input unit and a system for processing an audio signalsupplied from the audio input unit are not illustrated in FIG. 1, inputaudio data is supplied to the compression/expansion unit 44 via thememory 40. The compression/expansion unit 44 carries out compressing,coding, and multiplexing processing on the input image and audio data,based on a moving image compression method such as moving pictureexperts group (MPEG) 2 or H.264. The compression/expansion unit 44outputs the data in a predetermined format.

A recording medium control unit 46 writes the image and audio datacompressed by the compression/expansion unit 44 in a recording medium48. Examples of the recording medium 48 include a nonvolatilesemiconductor memory such as a flash memory, a hard disk apparatus, anda recordable optical disk.

Based on instructions from the system control unit 50, the cameracontrol unit 30 controls operations of the lens control unit 24, theexposure control unit 20, the image sensor driving unit 28, and thecamera signal processing unit 36.

The system control unit 50 includes a central processing unit (CPU) andcontrols individual units according to instructions from an operationunit 52 operated by a user, setting values, or an operation status. Forexample, the system control unit 50 controls an imaging operation viathe camera control unit 30 and controls recording/reproducing data inthe recording medium 48 via the recording medium control unit 46.Needless to say, the system control unit 50 comprehensively controls theentire imaging apparatus by executing predetermined programs.

An external interface (I/F) 54 sends and receives signals in apredetermined format to and from an external device connected to anexternal terminal 56. Examples of the external terminal 56 includeterminals that comply with standards such as the institute of electricaland electronic engineers (IEEE) 1394, universal serial bus (USB) and/orthe high-definition multimedia interface (HDMI). Examples of the signalformat include analog video signals (audio/video) and digital videosignals of various resolutions and frame rates.

The operation unit 52 is used as a user interface and includes a shutterbutton, a zoom button, a power-source on/off button, and a recordstart/stop button used during imaging or recording. The operation unit52 also includes a select button, a determination button, and a cancelbutton used to make selection on the menu, set parameters, and the like.

A display control unit 58 drives a display apparatus 60, which is usedas a monitor displaying captured images and reproduced images and isused as a display unit displaying various types of managementinformation to users.

The memory 40 is shared by each function block via the data bus 38, andis formed by a read only memory (ROM) and/or a random-access memory(RAM). Each function block connected to the data bus 38 can write andread data in and from the memory 40 via a memory control unit 42.

Image and audio data recorded in the recording medium 48 is reproducedas follows. The recording medium control unit 46 supplies compresseddata read from the recording medium 48 to the compression/expansion unit44. The compression/expansion unit 44 decompresses the input compresseddata to restore video data and audio data.

The display control unit 58 supplies the restored video data to thedisplay apparatus 60, which displays a reproduced image using thesupplied video data. Examples of the display apparatus 60 include aliquid crystal display panel and an organic light-emitting device. Aspeaker (not illustrated) outputs reproduced audio. When necessary,reproduced video data and audio data is output to the outside via theexternal I/F 54 and the external terminal 56.

An operation for controlling the frame rate at which video data isrecorded in the recording medium 48 will be described below. FIG. 2illustrates a relationship between captured frames temporarily stored inthe memory 40 and recorded frames recorded in the recording medium 48.FIG. 2A illustrates a zoom operation period during imaging carried outby the operation unit 52. FIG. 2B illustrates captured framestemporarily stored in the memory 40. FIG. 2C illustrates recorded framesrecorded in the recording medium 48.

FIG. 2A illustrates a zoom-on period, which is when a zoom operation iscarried out by a photographer, and a zoom-off period, which is when nozoom operation is carried out by a photographer. As illustrated in FIG.2B, the camera control unit 30 drives the image sensor 26 at a rate of1/240 second, irrespective of on/off of the zoom operation. The imagedata of each frame is temporarily stored in the memory 40 at a framerate of 1/240 second.

The camera control unit 30 controls the recording medium control unit 46to write frame images temporarily stored in the memory 40 in therecording medium 48, as illustrated in FIG. 2C. More specifically, thenumber of the frame images captured during the zoom-off period andtemporarily stored in the memory 40 is thinned to ¼, and the resultantframe images are written in the recording medium 48 at a normal framerate of 1/60 second. In the zoom-on period, the frame images temporarilystored in the memory 40 at a frame rate of 1/240 second are written inthe recording medium 48 without change.

FIG. 3 is a flow chart illustrating a control operation of a recordingframe rate. The system control unit 50 monitors a zoom operation carriedout by the operation unit 52. In step S1, if the system control unit 50detects a zoom operation during a recording operation (YES in step S1),the system control unit 50 carries out the processing in steps S2 to S5.If not (NO in step S1), the processing proceeds to step S6.

If the system control unit 50 detects a zoom operation (YES in step S1),in step S2, the system control unit 50 determines the zoom speed. Forexample, the zoom speed is determined based on the force applied by theuser to the zoom switch on the operation unit 52 or the length of timewhile the zoom switch is pushed. If a zoom speed is previously orinitially set on the menu screen or the like, the system control unit 50refers to the zoom speed. In other words, the system control unit 50also functions as a zoom speed detection unit.

In step S3, based on the zoom speed determined in step S2, the systemcontrol unit 50 sets a recording frame rate. FIG. 4 illustrates arelationship between the recording frame rate and the zoom speed. Asillustrated in FIG. 4, a higher recording frame rate is set for a higherzoom speed, and a lower recording frame rate is set for a lower zoomspeed.

As the relationship between the zoom speed and the recording frame rateof FIG. 4 illustrates, the recording frame rate may be changed on astep-by-step basis depending on the zoom speed. Alternatively, therecording frame rate may be changed linearly. Such a relationshipbetween the zoom speed and the recording frame rate is previously set ina form of a table and stored in the memory 40 (ROM therein).

In step S4, the system control unit 50 carries out a thinning process onthe image frame data in the memory 40 based on the recording frame rateset in step S3, and records the remaining frames in the recording medium48. In step S5, the system control unit 50 records zoom magnificationinformation and frame rate management information in the recordingmedium 48 as metadata, along with video data of each frame. In step S6,if instructions to stop recording are input, the processing ends. Ifnot, steps S1 to S5 are repeated.

Since the recording frame rate is changed based on the zoom speed, whenimages are captured at a low zoom speed, the images are recorded at alow frame rate. Thus, since unnecessary captured frames are notrecorded, consumption of the available capacity of the recording medium48 can be suppressed. While an object moves rapidly during a zoom-onperiod at a high zoom speed, since the object is followed at a highframe rate, captured scenes in which the angle of view changes greatlycan be recorded with good quality at a high sampling rate.

FIG. 5 illustrates a structure of metadata added to the video data ofeach frame, and data examples. The video data includes a header portion,a metadata portion, and a data portion. The header portion stores basicinformation that does not change depending on time, such as imagingapparatus information, format information, and thumbnail information.

The metadata portion stores information that is updated depending onvarious imaging conditions, such as a date, time, a recording framerate, and a shutter speed. The data portion stores image data and audiodata encoded in a predetermined format. The metadata is embedded in thevideo data in the metadata format using a description language such asextensible markup language (XML) or hypertext markup language (HTML).The metadata may be added as binary data or a watermark.

FIG. 6 schematically illustrates a relationship between the recordingframe rate and the zoom magnification according to the present exemplaryembodiment. In FIG. 6, the horizontal axis represents time, and thevertical axis represents zoom magnification (angle of view of theimage). Double circles represent frames captured at 1/240-secondintervals, and single circles represent frames captured at 1/60-secondintervals. According to the present exemplary embodiment, the framesrepresented by the single circles and captured at 1/60-second intervalsare recorded in the recording medium 48, irrespective of on/off of thezoom operation.

In the example illustrated in FIG. 6, during the zoom operation, namely,during the period between time t3 and time t9 when the zoommagnification changes, frames captured at 1/240-second intervals, whichis a rate higher than normal rate, are recorded in the recording medium48. Namely, captured frames are recorded at 1/240-second intervalsduring the zoom operation, and when the zoom operation is not carriedout, the captured frames are thinned and recorded at 1/60-secondintervals.

To ensure compatibility for reproduction, according to the presentexemplary embodiment, storage of the video data in the recording medium48 is controlled as follows. FIG. 7 illustrates an example of how imagedata illustrated in FIG. 6 is allocated and recorded in the recordingmedium 48.

Frames captured at t1, t2, t3, and the like at a normal recording framerate of 1/60 second are stored in a main storage area A of the recordingmedium 48. On the other hand, if a zoom operation is carried out and theframe rate is changed, the frames captured at a higher frame ratebetween t3, t3a, t3b, t3c, t4, . . . , and t9 are stored in asub-storage area Sub of the recording medium 48.

Thus, by separating video data depending on the frame rate andseparately storing the data in different areas of the recording medium48, even when an apparatus without special reproduction functions isused, the video data stored in the main storage area A can bereproduced. Thus, compatibility for reproduction can be ensured. Themain storage area A and the sub-storage area Sub are distinguishedlogically in terms of file management, and thus physical recordinglocations of these areas may be identical.

In FIG. 7, the frames captured at t3, t4, t5, and the like are recordedin the sub-storage area Sub as well as in the main storage area A. Thisis for decryption processing carried out during reproduction. If theframes captured at t3, t4, t5, and the like in the main storage area canbe used, these frames do not need to be stored in the sub-storage areaSub.

A reproduction operation of the present exemplary embodiment will bedescribed below. According to the present exemplary embodiment, beforereproducing videos of different frame rates, an interpolation orthinning process is carried out. In this way, scenes captured during azoom operation can be provided with special reproduction effects. FIG. 8is a flow chart illustrating a reproduction operation.

In step S11, based on reproduction instructions from the operation unit52, the system control unit 50 reads metadata about a specified videofile from the file management information of the recording medium 48. Instep S12, the system control unit 50 refers to the metadata anddetermines whether videos of different frame rates are recorded in thevideo file to be reproduced.

If images are not recorded at different recording frame rates (NO instep S12), the processing proceeds to step S16. In step S16, normalreproduction processing is carried out.

If images are recorded at different recording frame rates (YES in stepS12), in step S13, the system control unit 50 reads special reproductioneffects previously set by the user. In steps S14 and S15, the systemcontrol unit 50 controls displayed frames and voice output as describedbelow. FIG. 9 is a table illustrating special reproduction effects(modes) that involve zoom operations and the effects of the individualmodes.

A linear smoothing mode (mode 1) is a reproduction interpolation mode inwhich, even when the speed of a zoom operation carried out by the useris not constant, the zoom speed is changed during reproductionprocessing and reproduced images are displayed (rendered) at a constantzoom speed. In the linear smoothing mode, among the frames recorded inthe recording medium 48, the system control unit 50 selects frames sothat the zoom magnification of the selected frames changesproportionally. Next, the selected frames are displayed on the displayapparatus 60 at a normal frame rate.

FIG. 10 schematically illustrates frames displayed in the linearsmoothing mode. The horizontal axis represents time and the verticalaxis represents zoom magnification (angle of view of the image). Doublecircles represent frames recorded at 1/240-second intervals during azoom operation, and single circles represent frames recorded at1/60-second intervals. During the period between time t3 and time t9when the zoom magnification is changing, captured frames are recorded at1/240-second intervals.

When the zoom operation is not carried out, the captured frames arethinned and recorded at a frame rate of 1/60-second intervals. Asillustrated in FIG. 10, the zoom speed is not constant during imaging,i.e., the zoom speed fluctuates during a zoom operation from the wideend to the telephoto end.

The linear smoothing mode will be described in detail with reference toFIG. 10. Actually recorded frames are shifted so that the zoommagnification linearly changes as set in the linear smoothing mode, andthe frames used to be reproduced and displayed at a normal frame rateare selected and determined. In FIG. 10, the frames at time t5c, t6c,t7c, t8a, and t8c are selected to be displayed.

The frame at time t5c is shifted as a frame at time t4 (frame 70-1represented as a black circle) to be displayed. The frame at time t6c isshifted as a frame at time t5 (frame 70-2 represented as a black circle)to be displayed. The frame at time t7c is shifted as a frame at time t6(frame 70-3 represented as a black circle) to be displayed. The frame attime t8a is shifted as a frame at time t7 (frame 70-4 represented as ablack circle) to be displayed. The frame at time t8c is shifted as aframe at time t8 (frame 70-5 represented as a black circle) to bedisplayed.

Thus, in the linear smoothing mode, captured frames (recorded frames)whose zoom magnification changes linearly from the wide end to thetelephoto end are re-arranged on the time axis so that the frames aredisplayed at a constant zoom speed.

A dynamic shooting mode (mode 2) is a reproduction interpolation mode inwhich an image is zoomed rapidly as the object comes closer,irrespective of the zoom operation speed by the user. In other words,the linear smoothing mode provides a zoom effect of gradually increasingthe zoom speed.

Among the recorded frames in the recording medium 48, the system controlunit 50 selects frames so that the speed, at which the zoommagnification changes, is gradually increased. The selected frames aredisplayed on the display apparatus 60 at a normal frame rate.

FIG. 11 schematically illustrates frames displayed in the dynamicshooting mode. The horizontal axis represents time and the vertical axisrepresents zoom magnification (angle of view of the image). Doublecircles represent frames recorded at 1/240-second intervals during azoom operation, and single circles represent frames recorded at1/60-second intervals.

During the period between time t3 and time t9 when the zoommagnification changes, captured frames are recorded at 1/240-secondintervals. When the zoom operation is not carried out, the capturedframes are thinned and recorded at a frame rate of 1/60-secondintervals. In the example illustrated in FIG. 11, the speed, at whichthe zoom magnification changes, is constant during imaging.

The dynamic shooting mode will be described in detail with reference toFIG. 11. Actually recorded frames are shifted so that the zoommagnification is changing in a curve as set in the linear smoothingmode, and the frames used to be reproduced and displayed at a normalframe rate are selected and determined. In FIG. 11, the frames at timet3, t3a, t3b, t4, and t5a are selected to be displayed.

The frame at time t3 is temporally shifted as a frame at time t4 (frame72-1 represented as a black circle) to be displayed. The frame at timet3a is temporally shifted as a frame at time t5 (frame 72-2 representedas a black circle) to be displayed. The frame at time t3b is temporallyshifted as a frame at time t6 (frame 72-3 represented as a black circle)to be displayed. The frame at time t4 is temporally shifted as a frameat time t7 (frame 72-4 represented as a black circle) to be displayed.The frame at time t5a is temporally shifted as a frame at time t8 (frame72-5 represented as a black circle) to be displayed.

Thus, in the dynamic shooting mode, frames whose zoom magnificationchanges nonlinearly from the wide end to the telephoto end are displayedat a normal frame rate, so that the zoom speed is gradually increased.

A soft landing mode (mode 3) is a reproduction interpolation mode inwhich an image is zoomed in a decreasing speed as the object comescloser, irrespective of the zoom operation speed by the user. Namely,the soft landing mode provides a zoom effect of gradually decreasing thezoom speed.

Among the recorded frames in the recording medium 48, the system controlunit 50 selects frames so that the zoom speed gradually decreases as theobject comes closer. The selected frames are displayed on the displayapparatus 60 at a normal frame rate.

FIG. 12 schematically illustrates frames displayed in the soft landingmode. The horizontal axis represents time and the vertical axisrepresents zoom magnification (angle of view of the image). Doublecircles represent frames recorded at 1/240-second intervals during azoom operation, and single circles represent frames recorded at1/60-second intervals.

During the period between time t3 and time t9 when the zoommagnification changes, captured frames are recorded at 1/240-secondintervals. When the zoom operation is not carried out, the capturedframes are thinned and recorded at a frame rate of 1/60-secondintervals. In the example illustrated in FIG. 12, the speed, at whichthe zoom magnification is changing, is constant during imaging.

The soft landing mode will be described in detail with reference to FIG.12. Actually recorded frames are shifted so that the zoom magnificationchanges in a curve as set in the soft landing mode, and the frames usedto be reproduced and displayed at a normal frame rate are selected anddetermined. In FIG. 12, the frames at time t6c, t8, t8b, t8c, and t9 areselected to be displayed.

The frame at time t6c is temporally shifted as a frame at time t4 (frame74-1 represented as a black circle) to be displayed. The frame at timet8 is temporally shifted as a frame at time t5 (frame 74-2 representedas a black circle) to be displayed. The frame at time t8b is temporallyshifted as a frame at time t6 (frame 74-3 represented as a black circle)to be displayed. The frame at time t8c is temporally shifted as a frameat time t7 (frame 74-4 represented as a black circle) to be displayed.The frame at time t9 is temporally shifted as a frame at time t8 (frame74-5 represented as a black circle) to be displayed.

Thus, in the soft landing mode, frames whose zoom magnification changesnonlinearly from the wide end to the telephoto end are interpolated anddisplayed, so that the zoom speed is gradually decreased.

A slow motion mode (mode 4) is a reproduction mode in which all theframe images recorded at a high frame rate during a zoom operation arerendered at a preset magnification. For example, frame images capturedand recorded at a high frame rate of 1/240-second intervals during azoom period are reproduced at a normal frame rate of 1/60-secondintervals. As a result, the images are displayed at a slow speed (¼ ofthe original speed). Namely, the video recorded during a zoom operationis reproduced slowly and can be observed easily.

A skip mode (mode 5) is a reproduction mode that may be used when a usermakes a mistake in a zoom operation. For example, when a user increasesthe zoom magnification excessively and decreases hurriedly, sucherroneous operation is automatically detected, and reproduction of theimages captured by the erroneous operation is skipped. When a zoom-inoperation or a zoom-out operation is repeated in a short period of timeand an erroneous operation is made, this mode is effective in removingthe images captured by the erroneous operation.

FIG. 13 schematically illustrates frames displayed in the skip mode. Thehorizontal axis represents time and the vertical axis represents zoommagnification (angle of view of the image). Double circles representframes recorded at 1/240-second intervals during a zoom operation, andsingle circles represent frames recorded at 1/60-second intervals.

During the period between time t3 and time t9 when the zoommagnification is changing, captured frames are recorded at 1/240-secondintervals. When the zoom operation is not carried out, the capturedframes are thinned and recorded at a frame rate of 1/60-secondintervals. In the example illustrated in FIG. 13, the zoom magnificationchanges at a constant speed from the wide end at time t3 to thetelephoto end at t6. However, after time t6, the zoom magnification issomewhat decreased toward the wide end. In FIG. 13, the frames capturedduring the erroneous operation are not displayed on the screen.

The skip mode will be described in detail with reference to FIG. 13. Inthe skip mode, changes in an intermediate area in the zoom magnificationare ignored, and the skip mode sets such changes in the zoommagnification as illustrated in FIG. 13. More specifically, the startingpoint and the end point of a zoom operation are connected based on acertain rule (a straight line in FIG. 13). Next, actually recordedframes are shifted to correspond to the zoom magnification set in theskip mode, and the frames used to be reproduced and displayed at anormal frame rate are selected and determined.

In FIG. 13, the frames at time t3b, t4, t4b, and t5 are selected to bedisplayed. The frame at time t3b is temporally shifted as a frame attime t4 (frame 76-1 represented as a black circle) to be displayed. Theframe at time t4 is temporally shifted as a frame at time t5 (frame 76-2represented as a black circle) to be displayed. The frame at time t4b istemporally shifted as a frame at time t6 (frame 76-3 represented as ablack circle) to be displayed. The frame at time t5 is temporallyshifted as a frame at time t7 (frame 76-4 represented as a black circle)to be displayed.

Thus, when a user carries out an erroneous zoom operation or adjustment,if the user captures a video in which the zoom magnification isovershot, the user can reproduce and display the video without theovershoot portion by using the skip mode.

In the case of the example illustrated in FIG. 13, by monitoring theamount of the zoom operation within a certain time based on the metadataand determining an average zoom magnification A or B during a certainperiod, the zoom operation period that deviates from the average zoommagnification (overshoot portion) can be removed. Further, frameinterpolation during a zoom operation is carried out by re-arranging theframes during the zoom operation so that the changes of the zoommagnification are constant between the average zoom magnification A andB.

While five special reproduction modes have thus been described, any oneof these special reproduction modes may be specified for each scene as adisplay attribute. A plurality of these special reproduction modes maybe used in combination. Further, the above special zoom reproductioneffects have been described based on examples where a video is zoomed infrom the wide end to the telephoto end. However, needless to say, thesame processing is possible when a video is zoomed out from thetelephoto end to the wide end.

In step S15, the system control unit 50 continuously controls,reproduces, and outputs recorded audio, irrespective of selection ofreproduced and output frames. For example, the system control unit 50appropriately adjusts the time axis and/or deletes unnecessary portions(silent or prolonged portions, for example).

Referring back to FIG. 8, after these output frames and audio arecontrolled as described above, in step S16, the reproduction andoutputting processing is carried out. When necessary, the external I/F54 supplies the reproduction signal to the external device connectedthereto.

In step S17, if the system control unit 50 receives instructions to stopreproduction from the operation unit 52 or completes reproduction of theobject to be reproduced (YES in step S17), the system control unit 50ends the reproduction processing. If not (NO in step S17), theprocessing returns to step S11.

FIG. 14 is a block diagram illustrating functions of the system controlunit 50.

The system control unit 50 is connected to the operation unit 52 andalso to the compression/expansion unit 44, the recording medium controlunit 46, the memory 40, and the memory control unit 42 via the data bus38. The system control unit 50 includes a zoom operation detection unit50-1, a frame rate control unit 50-2, and a file management datageneration unit 50-3 as a recording system. Further, the system controlunit 50 includes a file management data detection unit 50-4, a displayframe control unit 50-5, and an audio control unit 50-6 as a reproducingsystem.

During recording, the zoom operation detection unit 50-1 detects whethera zoom operation is carried out by the user and the zoom speed during azoom operation. The zoom operation detection unit 50-1 supplies thedetected zoom operation information to the frame rate control unit 50-2.The frame rate control unit 50-2 controls the recording frame rate,based on the zoom operation information supplied from the zoom operationdetection unit 50-1.

The file management data generation unit 50-3 stores frame rate and zoomoperation information about video data stored in the recording medium 48in the file management data as metadata. Needless to say, the filemanagement data generation unit 50-3 may store the recording frame rateand zoom operation information in another file different from an imagefile, as long as each frame of video data is associated with therecording frame rate and zoom operation information.

During reproduction, the file management data detection unit 50-4 readsthe metadata about video data to be reproduced from the recording medium48, and supplies the metadata to the display frame control unit 50-5 andthe audio control unit 50-6. The display frame control unit 50-5controls display frames as described above, based on the metadatasupplied from the file management data detection unit 50-4 and aspecified reproduction mode. Similarly, the audio control unit 50-6controls output audio as described above, based on the metadata suppliedfrom the file management data detection unit 50-4 and a specifiedreproduction mode.

As described above in detail, according to the present exemplaryembodiment, frames captured during a zoom operation are recorded at ahigh frame rate, and during reproduction, based on a specified effect,certain frames are re-arranged on the time axis. In this way, videosduring zoom operations can be reproduced and displayed with variousdisplay effects.

Since videos are captured at a high frame rate, frame images of adesired zoom magnification can be extracted, and high-quality and smoothinterpolation can be realized. Additionally, even when a video capturedduring a zoom operation is visually undesirable because of a human erroror the like during the zoom operation, by carrying out an interpolationprocess, the video can be reproduced without the undesirable portion. Byseparately processing the video and audio during such interpolationprocess, while some frames are skipped in the video, continuity of theaudio can be maintained and reproduced without a break.

In the above exemplary embodiments, the camera unit constantly capturesframes at a high frame rate, and the frame rate at which the frames arerecorded in the recording medium 48 is decreased when necessary.However, the present invention is not limited to such example. Forexample, by changing the frequency of a timing signal output from theimage sensor driving unit 28, even when the frame rate at which thecamera unit captures frames is changed more flexibly, similar effectscan be obtained.

While a video captured during a zoom operation often includes importantscenes, similarly, videos captured before and after a zoom operationoften include important scenes. Thus, irrespective of on/off of a zoomoperation, frames are recorded at a high frame rate, to record videodata captured during certain periods before and after the zoom operationin the recording medium 48 at a high frame rate. During reproduction,special reproduction effects similar to those of the first exemplaryembodiment are applied to a zoom operation period and the periods beforeand after the zoom operation period. In this way, a video can bereproduced smoothly during a zoom operation period and the periodsbefore and after the zoom operation period.

FIG. 15 illustrates a reproduction operation of the present exemplaryembodiment. The horizontal axis represents time.

FIG. 15A illustrates on/off of a zoom operation during recording. FIG.15B illustrates a frame control signal indicating a special reproductionperiod formed by a zoom operation period and the periods before andafter the zoom operation period. Thus, the special reproduction periodis formed by the zoom operation period illustrated in FIG. 15A, a periodD1 before the zoom operation period, and a period D2 after the zoomoperation period. FIG. 15C illustrates captured frames. FIG. 15Dillustrates frames recorded in the recording medium 48.

As seen from FIGS. 15B and 15D, according to the present exemplaryembodiment, in addition to the frames captured during a zoom operation,the frames captured during periods D1 and D2, which are before and afterthe zoom operation, are also recorded in the recording medium 48 at ahigh frame rate of 1/240-second intervals. The lengths of periods D1 andD2 may be different from each other.

To record frames captured during period D1, which is before the zoomoperation, in the recording medium 48, a buffer memory capable ofstoring video data output from the camera signal processing unit 36 formore than period D1 is set in the memory 40 in advance.

If the start of a zoom operation is detected, frames captured duringperiod D1, the zoom operation, and period D2 and recorded in the buffermemory 40 are encoded without changing the frame rate, and recorded inthe recording medium 48. If the start of a zoom operation is notdetected, frames captured after period D1 and stored in the buffermemory 40 are thinned at a frame rate of 1/60-second intervals. Theresultant frames are then encoded and recorded in the recording medium48.

FIG. 16 schematically illustrates a relationship between the recordingframe rate and zoom magnification according to the present exemplaryembodiment. The horizontal axis represents time and the vertical axisrepresents zoom magnification (angle of view of the image).

Double circles represent frames captured at 1/240-second intervals, andsingle circles represent frames captured at 1/60-second intervals.According to the present exemplary embodiment, the frames represented bythe single circles and captured at 1/60-second intervals are recorded inthe recording medium 48, irrespective of on/off of the zoom operation.In addition to the period between time t3 and time t9 when the zoommagnification changes, video data captured during the period betweentime t1 and time t3, which is before the zoom operation, and during theperiod between time t9 to time t11, which is after the zoom operation,is also recorded in the recording medium 48 at a frame rate of1/240-second intervals, which is higher than normal rate.

Thus, according to the present exemplary embodiment, in addition to azoom operation period, video is recorded in the recording medium 48 at ahigh frame rate during certain periods before and after the zoomoperation period. Generally, when a user carries out a zoom operation,video captured before and after a zoom-in operation or a zoom-outoperation includes a target object. Thus, the video includes importantscenes that reflect the photographer's intention, and examples of suchscenes include goal scenes in sports games and close-up facialexpressions. By recording the video captured before and after a zoomoperation at a high frame rate as described above, the video capturedbefore and after the zoom operation can be displayed effectively.

While the above exemplary embodiments use an optical zoom, whichoptically zooms an optical image incident on the image sensor 26, thepresent invention may similarly be applicable to an imaging apparatus,which uses an electronic zoom that electronically zooms an image signalgenerated by the sensor 26.

The rate of a video signal has been described as a frame rate in theabove description. However, in the case of an interlace signal, byreplacing the frame rate with a field rate, similar effects can beobtained.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

What is claimed is:
 1. A method for reproducing, comprising: carryingout a thinning process on a video signal recorded at a predeterminedframe rate; reproducing the video signal; selecting a reproduction mode;and controlling changing a zoom effect during reproduction according tothe selected reproduction mode during a period when a zoom operationunit is operated during recording.
 2. The method according to claim 1,further comprising: controlling reading out zoom operation informationrecorded with the video signal; and determining, based on the zoomoperation information, whether the zoom operation unit is operatedduring recording.
 3. The method according to claim 2, wherein the zoomoperation information is zoom speed.
 4. A method for reproducing,comprising: carrying out a thinning process on a video signal recordedat a predetermined frame rate; reproducing the video signal; selecting azoom effect; and controlling the thinning process or selecting of framesto be displayed, during reproduction according to the selected zoomeffect during a period when a zoom operation unit is operated duringrecording.
 5. The method according to claim 4, further comprising:controlling reading out zoom operation information recorded with thevideo signal; and determining, based on the zoom operation information,whether the zoom operation unit is operated during recording.
 6. Themethod according to claim 5, wherein the zoom operation information iszoom speed.
 7. The method according to claim 4, wherein the thinningprocess is controlled during reproduction according to the selected zoomeffect during a period when a zoom operation unit is operated duringrecording, so as to control selecting of frames to be displayed.
 8. Themethod according to claim 4, further comprising reproducing the videosignal at a different frame rate.
 9. A method for reproducing,comprising: carrying out a thinning process on a video signal recordedat a predetermined frame rate; reproducing the video signal; andcontrolling selecting of frames to be displayed or thinned, duringreproduction, during a period when a zoom operation unit is operatedduring recording, so as to control changing of a zoom effect.
 10. Themethod according to claim 9, further comprising: controlling reading outzoom operation information recorded with the video signal; anddetermining, based on the zoom operation information, whether the zoomoperation unit is operated during recording.
 11. The method according toclaim 10, wherein the zoom operation information is zoom speed.
 12. Amethod for reproducing, comprising: carrying out a thinning process on avideo signal recorded at a predetermined frame rate; reproducing thevideo signal; selecting a reproduction mode; controlling the thinningprocess during reproduction according to the selected reproduction modeduring a period when a zoom operation unit is operated during recording;and changing a zoom effect or reproducing the video signal at adifferent frame rate.
 13. The method according to claim 12, furthercomprising: controlling reading out zoom operation information recordedwith the video signal; and determining, based on the zoom operationinformation, whether the zoom operation unit is operated duringrecording.
 14. The method according to claim 13, wherein the zoomoperation information is zoom speed.
 15. The method according to claim12, further comprising: controlling reading out zoom speed informationrecorded with the video signal; and determining, based on the zoom speedinformation, whether the zoom operation unit is operated duringrecording.
 16. A non-transitory computer-readable storage medium storinga program for causing a computer to execute a method for reproducing,the method comprising: carrying out a thinning process on a video signalrecorded at a predetermined frame rate; reproducing the video signal;selecting a reproduction mode; and controlling changing a zoom effectduring reproduction according to the selected reproduction mode during aperiod when a zoom operation unit is operated during recording.
 17. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute a method for reproducing, the methodcomprising: carrying out a thinning process on a video signal recordedat a predetermined frame rate; reproducing the video signal; selecting azoom effect; and controlling the thinning process or selecting of framesto be displayed, during reproduction according to the selected zoomeffect during a period when a zoom operation unit is operated duringrecording.
 18. A non-transitory computer-readable storage medium storinga program for causing a computer to execute a method for reproducing,the method comprising: carrying out a thinning process on a video signalrecorded at a predetermined frame rate; reproducing the video signal;and controlling selecting of frames to be displayed or thinned, duringreproduction, during a period when a zoom operation unit is operatedduring recording, so as to control changing of a zoom effect.
 19. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute a method for reproducing, the methodcomprising: carrying out a thinning process on a video signal recordedat a predetermined frame rate; reproducing the video signal; selecting areproduction mode; controlling the thinning process during reproductionaccording to the selected reproduction mode during a period when a zoomoperation unit is operated during recording; and changing a zoom effector reproducing the video signal at a different frame rate.